Skip to content

Some websites using Cloudflare CAPTCHA require the challenge again after restarting a script (even with custom profile) #1455

Open
@baptx

Description

@baptx

Some websites using Cloudflare CAPTCHA require the CAPTCHA challenge again after restarting a script, even with a custom profile. The issue happened both with old version 3.5.0 and new version 3.5.2 of undetected-chromedriver.
A website like nowsecure.nl will not ask the CAPTCHA again when restarting the script with a custom profile where the CAPTCHA was completed, however a website like doctolib.fr will always ask it (which is not always the case without WebDriver). Any idea why?
You can find a test script below.

import undetected_chromedriver as uc
import time

driver = uc.Chrome(user_data_dir="/tmp/undetected-chromedriver")

# need to to open in new tab to complete CAPTCHA (https://github.com/ultrafunkamsterdam/undetected-chromedriver/issues/1388#issuecomment-1651886887)
driver.execute_script('window.open("http://nowsecure.nl/#relax");')
#driver.execute_script('window.open("https://www.doctolib.fr");')
time.sleep(5) # wait until page has loaded
#driver.switch_to.window(window_name=driver.window_handles[0]) # switch to first tab (should not be necessary)
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0])

time.sleep(30)

Activity

baptx

baptx commented on Aug 9, 2023

@baptx
Author

I noticed with a normal web browser without WebDriver, the CAPTCHA will be asked again on a website like doctolib.fr if we refresh the web page using Ctrl+Shift+R instead of Ctrl+R or F5. The difference with Ctrl+Shift+R is that it sends the "Pragma: no-cache" and "Cache-Control: no-cache" HTTP request headers. Is there a way to avoid that (without manually editing request headers using an addon) when restarting a WebDriver script, like it is working with nowsecure.nl?

Update: I tried removing request headers like Pragma and Cache-Control with the addon Header Editor on Firefox and Chromium but they are still sent (empty value on Chromium).
The interesting thing is that when pressing Ctrl+Shift+R on Chromium without WebDriver, the CAPTCHA is not asked again unlike Firefox. And I can see the 2 headers mentioned previously are sent.

baptx

baptx commented on Aug 9, 2023

@baptx
Author

Workaround: we can automate the CAPTCHA validation with the following code (you can change the sleep time if necessary but the best would be to avoid the sleep by knowing when the CAPTCHA finished loading, maybe with MutationObserver).

time.sleep(5)
driver.find_element(By.ID, "challenge-stage").click()

The advantage is that we don't need a custom profile now. But it would be good to have a solution for the CAPTCHA issue that is asked again with a custom profile.

Update: in fact you may not need the sleep with the following code.

el = WebDriverWait(driver, timeout=10).until(lambda d: d.find_element(By.ID,"challenge-stage"))
el.click()

But in some cases, only the first workaround with the sleep will work, which would mean that waiting to find the ID challenge-stage does not mean the CAPTCHA is loaded (waiting for the element to be clickable did not fix the issue: https://stackoverflow.com/questions/26943847/check-whether-element-is-clickable-in-selenium/26943922#26943922).

NCLnclNCL

NCLnclNCL commented on Aug 10, 2023

@NCLnclNCL

Use chrome 102 can bypass easy

Krylanc3lo

Krylanc3lo commented on Aug 10, 2023

@Krylanc3lo

@NCLnclNCL I tried 102 but it didn't work. you are using version_number = 102 and download and downgrade chrome browser to 102?

AntidetectBrowser

AntidetectBrowser commented on Aug 10, 2023

@AntidetectBrowser

@NCLnclNCL I tried 102 but it didn't work. you are using version_number = 102 and download and downgrade chrome browser to 102?

use this #1456 (comment)

baptx

baptx commented on Aug 10, 2023

@baptx
Author

@NCLnclNCL I prefer to avoid old versions of web browsers but I will try it if no other workaround is working in the future.
It would be good to know why it works on version 102, so we can try to have the fix on the latest version of Chromium.

baptx

baptx commented on Aug 10, 2023

@baptx
Author

@AntidetectBrowser Do you have the source code of the chromedriver.exe file from your repository available somewhere? This would be good so we can see the patch you used and avoid viruses by compiling ourselves. Also the exe file is for Windows only, not Linux.

teamothyz

teamothyz commented on Aug 29, 2023

@teamothyz

do you have the solution for this? I'm still looking for

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

      Participants

      @baptx@Krylanc3lo@teamothyz@NCLnclNCL@AntidetectBrowser

      Issue actions

        Some websites using Cloudflare CAPTCHA require the challenge again after restarting a script (even with custom profile) · Issue #1455 · ultrafunkamsterdam/undetected-chromedriver